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1 Introduction 

Dynamical Lefschetz trace formulas for flows on compact manifolds were first 
conjectured by Guillemin [0 and later but independently by Patterson 
These formulas express the alternating sum of the traces of the induced flow 
on cohomology by a sum over contributions from the compact orbits. 

Here the cohomology theory depends on the choice of a foliation which is 
respected by the flow. 

This note makes a contribution to Patterson's setup. He looks at Anosov 
flows and considers the unstable foliation. An interesting example of this 
situation is provided by the geodesic flow on the sphere bundle of a cocompact 
quotient of PSL2(M) by a lattice. Using representation theory, Patterson was 
able to define a trace on the infinite dimensional foliation cohomologies. The 
dynamical Lefschetz trace formula was then shown to be a consequence of 
the Selberg trace formula. Incidentally this example also fits into Guillemin's 
framework who treated it in Q. 

Much more work in the context of more general symmetric spaces was done 
by Bunke, Deitmar, Juhl, Olbrich and Schubert. We refer the reader to the 
book of Juhl [0 for a comprehensive overview. 

Among all Anosov flows there are the ones with an integrable complement 
or equivalently, those where the sum of the stable and the unstable bundle is 
integrable. One gets examples by suspending Anosov diffeomorphisms and 
conjecturally all Anosov flows with an integrable complement should arise in 
this way up to a time change by a constant factor, |Q §3. 

Granting this conjecture the dynamical Lefschetz trace formula for such 
Anosov flows can be reduced to the following problem: 

Let / be an Anosov diffeomorphism of a compact manifold X whose unstable 



foliation JF" is smooth. Establish a dynamical Lefschetz trace formula of the 
following form: 

(1) ^(-l)PTr(/* I if^„(X)) = E - I ' 

where 

= sgn det(l - TJ I T^) . 

Here for a fixed point x, resp. is the subspace of T^X corresponding 
to the eigenvalues of T^f of absolute value smaller resp. greater than one. 

Note that since / is supposed to be Anosov, in particular the eigenvalue 
A = 1 does not occur on T^X. Hence all determinants are non-zero. The 
definition of H^u{X) can be found in §2 below. 

Formula ([l|) is a dynamical Lefschetz trace formula for the discrete dynamical 
system determined by /. 

The analytic difficulties regarding dynamical Lefschetz trace formulas for 
flows persist in formula (|I|) for diffeomorphisms. The main problem being 
that the complex defining H'jru{X) is not elliptic but only elliptic along the 
unstable leaves. In particular we cannot appeal to the work of Atiyah and 
Bott on Lefschetz trace formulas for endomorphisms of elliptic complexes. 



According to ||M[ §4 the following classification theorem for Anosov diffeo- 



morphisms is expected. After lifting to a finite covering any Anosov dif- 
feomorphism should become topologically conjugate to an algebraic Anosov 
diffeomorphism /. The latter come about as follows. Let G be a simply con- 
nected real nilpotent Lie group with a cocompact lattice F and consider an 
automorphism f of G with /(F) = F. Assume that the induced Lie algebra 
automorphism of Q does not have any eigenvalue of absolute value one. 
Then the induced diffeomorphism / of the compact nilmanifold X = T \ G 
is Anosov, and the Anosov diffeomorphisms obtained in this way are called 
"algebraic" . 

If we only demand that / is an endomorphism of G with /(F) C F and such 
that ft, does not have 1 as an eigenvalue then the induced map / of X will 
be called a generalized algebraic Anosov map. 
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In the present note we first prove formula (|T]) for algebraic Anosov diffeo- 
morphisms. In this case by a subtle argument we can calculate the reduced 
leafwise cohomology i.e. the maximal Hausdorff quotient of H^u{X). It turns 
out to be finite dimensional so that the definition of the trace of /* does not 
present a problem. 

Next, using basic aspects of the representation theory of q, we prove formula 
([^) also for generalized algebraic Anosov maps. In this case, even the reduced 
leafwise cohomology is infinite dimensional in general. Our definition of the 
trace of /* then depends in an essential way on representation theory. This 
seems to be the only method to define these traces. Namely the simplest 
non-trivial case of a generalized Anosov map / which is not Anosov is multi- 
plication by — 1 on M/Z. We are not aware of any purely functional-analytic 
definition of a trace which applies to the map /* on C°°(]R/Z). 

We would like to thank the referees whose comments have led to some im- 
provements of the paper. 



2 Background on leafwise cohomology 

For a smooth foliation JF on a closed smooth manifold X consider the de Rham 
complex of smooth real valued forms along the leaves: 

(2) (r(X,A-T*^),c/^). 
It is a complex of nuclear Frechet spaces. 

Here the exterior differential djr is defined by the same formula as the ordinary 
differential d. Integrability of TjF translates into the condition that = 0. 

The cohomology groups 

(3) H'^iX):=H-iTiX,A'T*J^),d^) 

are called the leafwise cohomology groups of the foliated manifold X. If TZ 
is the sheaf of smooth real valued functions on X which are locally constant 
on the leaves, we have 

(4) H-^ix)=H'ix,n). 
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The reduced leafwise cohomology is the maximal Hausdorff-quotient H^{X) 
of H^{X). It is the nuclear Frechet space obtained by dividing Keidjr by 
the closure of Imdjr. In general even the reduced cohomology is infinite 
dimensional, compare example |5.5| for example. 



Given another foliated manifold X', JF', any map f : X ^ X' which maps 
leaves of into leaves of JF' induces a continuous pullback map on cohomol- 
ogy 

r : H'^,iX') ^ H'^iX) . 

These maps pass to the reduced versions of leafwise cohomology. 

Consider now a smooth compact manifold X = r\G obtained as the quotient 
of a connected real Lie group G by a cocompact lattice T G G. For an 
immersed subgroup P C G with Lie algebra p C g consider the foliation JF 
of X with leaves TgP for g E G. Then we have a topological isomorphism 
with Lie algebra cohomology 

(5) H'^{X)^H'{p,C-'{X)). 

Indeed let L = p ® C°°{X) be the Lie algebra of vector fields tangent to the 
leaves. Then we have: 

r(X,A*T*^) = Homcoo(x)(Acoo(x)(^),C°°(X)) . 

This is an isomorphisms of complexes if the right hand side is equipped with 
the differential (3A.5) in ||Ml5|] Ch. Ill, Appendix. Formula (|^) follows. 



We now decompose the reduced leafwise cohomology using representation 
theory. The existence of the cocompact lattice F implies that Haar measure 
on G is unimodular. Let fi denote the unique G-invariant measure on X 
of volume one. Let R be the "regular" unitary representation of G on the 
complex Hilbert space L'^{X) = L'^{X,fi) by right translation: 

{RJ){rg') = f{rg'g). 

Since X is compact, the representation R decomposes into isotypic compo- 
nents H{tt) of finite multiplicity: 



(6) L\X) = 0i7( 
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Here G is the unitary dual of G, i.e. the set of equivalence classes of irre- 
ducible unitary representations of G. Each isotypic component is isomorphic 
to ® Vn where {pn, Vtt) is a fixed representation in the class tt and where 
M-^ is the finite dimensional multiplicity space, i.e. 

M,^RomG{V,,L\X)). 

We can now analyse the reduced leafwise cohomology using representation 
theory. Denote by H^{p, i?(7r)°°) the maximal Hausdorff quotient of 
HP{p, H{7r)°°), where H{7r)°° denotes the space of smooth vectors in H{'k). 

Lemma 2.1 The natural maps 

H^ip, H{%)'^) H^{X, C) = H^riX) C 

are split injections and hence have closed image. In this way the algebraic 
direct sum 

^H^ip^HinD 

TT 

becomes a dense subspace of 

HP,{X, C) = HU^) ® C . 

Every element of II^{X, C) can be written as an unconditionally convergent 
series ^^h^ with K e Hp{p,H{t:)'^). 

The proof is given below. It is based on an auxiliary result which requires 
the following notions: 

Definition 2.2 A sequence {Ei)i>i of non-trivial closed linear subspaces of 
a Frechet space E is called a Schauder basis of subspaces of E if the following 

conditions hold: 

a) Every element v E E can be written uniquely as a convergent series 

^ = YliLi '"i ^^^^ Vi e Ei. 

b) The projections E ^ Ei,v ^ Vi resulting from a) are continuous. 

If the series in a) converge unconditionally the basis {E^) is called uncondi- 
tional. 
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Sublemma 2.3 Let C be a complex of Frechet spaces with continuous dif- 
ferentials. Consider closed subcomplexes C' C C for i > 1 such that for 
every p the sequence (C,f)j>i is an unconditional Schauder basis of subspaces 
of . Then the natural maps 

are split injective and hence have closed image. They induce an inclusion 
with dense image 

i 

Every element of H^iC') can be written as an unconditionally convergent 
series Ylii^i ^^^^ ^« ^ H^i^D- 

Proof For any p the assumptions give continuous projections 
such that for all c in C^~^ we have 

i 

Since the differential is continuous we get 

dc = dj3i{c) . 

i 

On the other hand we have 

dc — Pi{dc) . 

i 

The representation of an element in as a convergent series of elements in 
the Cf is unique. Hence we find 

d(3i{c) ^ (3i{dc) 

for all c i.e. do Pi — Piod. Prom this and the continuity of Pi it follows that 

(7) Pi{dC^) C PiidCP-^) = dPiiCP-^) = dCr' . 
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The inclusion Kj : C' ^ C of complexes induces a continuous map on the 
p-th cohomology and hence a continuous map 

Ki : — > HP{C') . 

Similarly the continuous surjective map of complexes 

A : C* ^ C- 

leads to a continuous map 

A : H^'iC) — > H^{Cl) . 

We have (3ioKi = id and hence (3ioRi = id. Therefore split injection. 

Its image is closed since it is also the kernel of the continuous projector 
TTj = id — /tjo/?j on H^{C'). 

We now show that the induced map 

i 

is injective. So assume that the element ^Jq] goes to zero in H^^iC'). This 
means that Cj lies in dCP~^. Here q G Cf,dci = and almost all q are 
zero. Applying Pi we see that Cj lies in Pi{dCP~^) and by (|^) therefore in 
dCf~^. Thus [cj] = for all i. 

Next consider any element [c] in Hp{C'). We may write c as an uncondition- 
ally convergent series 

c = Cj with Cj = /3i(c) G Cf . 

i 

We have = Pi{dc) = d[3i{c) = dci. Hence [q] G H^iC') lies in the image of 
H^{C') and we have 

i 

in HP{C') the series being unconditionally convergent. In particular the 
algebraic direct sum of the spaces H'p{C') is dense in Hp{C'). □ 
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Proof of 271 Using the Sobolev embedding theorem one sees that the non- 
zero subspaces H{7r)°° C C°°{X) = L'^{X)°° form an unconditional Schauder 
basis of subspaces of C°°{X). It follows that the subcomplexes C'{q, H{7i)°°) 
of C'{q,C°°{X)) satisfy the assumptions of the sublemma. This implies 
lemma p.lL □ 



3 F-acceptable subgroups and leafwise coho- 
mology 

In this section we first prove that the reduced leafwise cohomology of certain 
homogenous foliations on nilmanifolds is isomorphic to finite dimensional Lie 
algebra cohomology. This result applies to the stable and unstable foliations 
of an algebraic Anosov diffeomorphism. It also applies to the foliation defined 
by a normal subgroup where it gives a result of Alvarez Lopez and Hector. 

Let G be a simply connected nilpotent Lie group with a cocompact lattice 
r C G. Let 

(8) G = Co^C,^...^Ck^ Ck+i = 1 

be the descending central series of G defined by C^+i = [G, Gj] and Gq = G. 
From the definition we see: 

3.1 The group Gj/Gj^i is central in G/Gj^i for every < j < A;. In partic- 
ular Gk is central in G. 



We will need the following basic results of Malcev ||Ma|] : 

3.2 The group = F fl Gj is a cocompact lattice in Gj for every j. 

3.3 The image F(j) of F in G/Gj is a cocompact lattice for every j. 



It follows from ^.2| and |3.^ that the image of F, in the abelian Lie algebra 



Gj/Gj^i is a cocompact lattice. The quotient is a real torus 

'^j = C'j/Cj+iFj = Tj \ Gj/Gj+i . 
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Definition 3.4 A closed connected subgroup P of G is called T -acceptable if 
the image of Pj = P (1 Cj in Aj is dense for every < j < k. 

In p.7| below we will see that the stable and unstable groups of an algebraic 
Anosov diffeomorphism oi X = T \ G are F-acceptable. Moreover, normal 
subgroups of G whose orbits in X are dense are F-acceptable by proposition 

Let be the foliation of X by the orbits of the right P-operation on X. The 
embedding of M into C°°{X) induces a map 

(9) H-{p,R) ^ H'{p,C^{X))i H'^X) . 

Theorem 3.5 If P is T -acceptable this is an isomorphism: 

H'{p,R) = H'^{X) . 

For the proof we require the following lemma. 

Lemma 3.6 Let p be a finite dimensional real Lie algebra and let 

= ao C . . . C a„ C p 

be a chain of normal subalgebras of p such that aj/aj-i is central in p/aj^i 
for every 1 < j < n. Let M be a complex vector space on which p acts C- 
linearly. We assume that an-i acts trivially and that an acts by a nontrivial 
character a. Then H'{p, M) = 0. 

Proof The Hochschild-Serre spectral sequence 

HP{p/an, H'^ian, M)) =^ H^+'^ip, M) 

shows that we may assume that p = a„. Clearly we may also assume that 
the inclusions in the chain 

= ao C . . . C a„ = p 
9 



are strict. Since = p acts non-trivially we have p 7^ and hence n > 1. 
The assertion is proved by induction on n. For n = 1 the Lie algebra p is 
abehan. Hence it operates on M- valued cocycles and therefore on if*(p,M) 
by multiplication with a c.f. [ |HS|| p. 591 below. On the other hand the 
operation of g on H'{q, N) is trivial for any Lie algebra q and any g-module 
A^. Since a was non-trivial, it follows that H'{p, M) = 0. 

Now fix some n > 2 and assume the assertion proven for n — 1. Consider the 
spectral sequence 

(10) H^ip/ai, H^iau M)) =^ H^+\p, M) . 

Since ai is central in p = a„ it follows that p acts by multiplication with a 
on M- valued cocycles of Oi and hence on H'^{ai, M) c.f. ||HS|| p. 591. Setting 
M' = H'^{ai, M) and a- = aj+i/ao, p' = p/ai we get a chain 

= a[, C . . . C a;_i = p' . 

Applying the induction hypotheses to the p'-action on M' gives 

i/*(p/ai, i7^(ai, M)) = H'{p', M') = . 

Using the spectral sequence (|10D we conclude that i7*(p, M) = 0. □ 

Proof of |3.5| It is enough to show that the natural inclusion C ^ C°°(X) = 
C°°(X, C) induces an isomorphism 

(11) ff-(p,C)^iy-(p,C°^(X)). 



According to ^]3| the image r(j) of F in G{j) = G/Cj is a lattice for every 
< J < A; + 1. The natural projection of right P-spaces 

X -^x, = rij)\GU) = r.c,\G 

induces a p-equivariant injection 

(12) C°^{Xj) ^ C°°{X) . 

We will show by descending induction on j that ([T^ induces isomorphisms 
of Frechet spaces for all < j < A; + 1 

(13) H'ip, C^iX^)) ^ H'ip, C^iX)) . 
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For j = k+1 this is clear. For the induction step fix < m < and assume 
that ( p!3D is an isomorphism for j = m + 1. Now consider the torus 

^ ^in C / C m+l^ m \ C*}Ti/Cm+l • 

The right G-action on X^+i induces a A-action on X^+i since the elements 
of Cm are central mod Cm+i- Fourier theory imphes that the x-isotypical 
components 

C~(X^+i)(x) 

of the induced A-action on C°^{Xm+i) form an unconditional Schauder basis 



of subspaces in the sense of definition ^T^. Here x runs over the characters 
of A. It follows that the subcomplexes 

c*(p,c~(x^+0(x)) 



of C°°(p, C°°(Xm+i)) satisfy the assumptions of sublemma p.3| . Hence there 
is a natural injection 

(14) 0iy*(p,C°°(X^+i)(x)) C ^•(p,c°°(x^+0) 



with dense image. Moreover the spaces H'{p,C°°{Xm+i){x)) are closed in 
H'{p, C°°(Xm+i)). We now apply lemma |3]^ to p = LieP and the chain 

= ao C . . . C a„ C p 

defined by Oj = Lie (P fl Cfc+i-j)- Here n = k + 1 — m. For M we take 
M = C°°{Xm+i){x) where x is any character of A which is non-trivial on 
Pm = PnCm- Then a„ = LiePm acts by the non-trivial character a = on 
M and an-i = Lie Pm+i acts trivially. It follows from the lemma that only 
X's with x{Pm) = 1 can contribute to the direct sum in ([T^). 

Now, by assumption P is F-acceptable. Hence the image of Pm in A is dense. 
The condition x{Pm) = 1 therefore implies that x is the trivial character 
X = 11. Thus P"(p,C°°(X„+i)(l)) is a dense subspace of 5"(p, C°°(X„+i)). 
Since it is also closed we get: 

(15) H'ip, C°"{Xm)) = H'ip, C^iXm+i)) . 
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Here we took into account that 

C°°(X„+i)(]l) = C°°(X™+i/A) = C°^(X„) . 

By the induction hypotheses we have an isomorphism 

iy*(p,C~(X^+i)) ^^•(p,C°°(X)) . 

Combining it with (|15|) we get the isomorphism ([13| ) for j = m. Note that a 
continuous algebraic isomorphism of Frechet spaces is necessarily a topologi- 
cal isomorphism by the open mapping theorem. Thus we have completed the 
induction step. The case j = of (|T3|) now give the assertion of the theorem. 

□ 

3.7 Let / be an algebraic Anosov diffeomorphism of a nilmanifold X = r\G 
as defined in the introduction. It is induced by an automorphism of G which 
we also denote by /. Let g'' C g be the subspace of all f G g for which 
/"(f) tends to zero as n — cx). This is the stable subspace of g. The 
unstable subspace of g is the space of all f of g such that f^{v) — > for 
n ^ — oo. Since / is a group automorphism it follows that /* is a Lie algebra 
automorphism and so g^ and g" are Lie subalgebras of g. Let and G" 
denote the corresponding connected subgroups of G. 



Proposition 3.8 In the situation of | J. Tf the groups P = G^ and P = G" are 
T-acceptable. 

Proof The stable subgroup of is the unstable subgroup of /. Hence it 
is enough to treat the case P = G"^. We have to show that for < j < k the 
image of Pj is dense in = \ Gj where Gj = Gj/Gj^i is abelian. We 
have Pj = Gj with respect to the induced automorphism fj of Gj. Hence it 
suffices to show that for abelian G the image of P = is dense in F \ G or 
equivalently that the image of F is dense in G/G^. 

So suppose G is abelian. Then the exponential map is an isomorphism of 
the Lie algebra of G with G, so G is a finite dimensional M-vector space. We 
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therefore rephrase the claim as follows. Let be a non-zero finite dimen- 
sional real vector space and let / be an automorphism of V with no complex 
eigenvalue of absolute value 1. Let be the stable subspace, i.e., 

V = {veV\ r{v) ^ as n ^ oo}. 

Let r C be a lattice with /(F) = F. Then we have to show that the image 
Tw of F is dense in W = V/V^ . This will follow if for every e- neighborhood 
U of zero in W the set U fl Tw contains a basis of W. Namely the lattice 
generated by such a basis is contained in F^^ and has a fundamental domain 
in U. Thus every given point in W is e-close to some point of this lattice 
and hence of Fvk- 

So, let U he a. neighborhood of zero in W. On W all eigenvalues of / are 
of absolute value > 1, therefore is contracting there. Let vi, . . . ,Vr he a. 
basis of W contained in T]y. Then the vectors /""(fi), . . . , /""("Wr) form a 
basis for every n G N and this basis shrinks to zero as n — oo, so the claim 
follows. □ 

Remark Indirectly the proof shows that 0. This is clear a priori since 
on the one hand | det/| = 1 because / respects a lattice and on the other 
hand / does not have eigenvalues of absolute value 1. 

Corollary 3.9 In the situation of ^. Tj we have natural isomorphisms 
H'^,{X) = H'{g',R) and H'^u{X) = H'{g'',R) . 

In particular these reduced leafwise cohomologies are finite dimensional. 



Proof This follows from proposition 3^ and theorem 3T5 using that JF'* and 



JF" are the foliations by the orbits of the G^-resp. G^-action on X. □ 

Remark We conjecture that for any Anosov diffeomorphism of a compact 
manifold with smooth stable and unstable foliations the reduced leafwise 
cohomologies with respect to these foliations are finite dimensional. 



3.10 In this final subsection we consider normal subgroups in simply con- 
nected real Lie groups. The following result holds: 
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Proposition 3.11 A normal subgroup P with TP dense in G is T-acceptable. 



Remark The proposition and theorem |3.5| imply the isomorphism 

H'^{X)^H'ip,R) . 

This result is due originally to Alvarez Lopez and Hector ||AH| Theorem 2.12 
with a different proof. 

Proof It suffices to show that TjPj is dense in Cj. We use induction on j. 
For j = this is the assumption. Assume the claim proven for j. Since 
Cj+i = [G, Gj] we pick x G G and y E Gj. Let 7nP„ be a sequence converging 
to X, where 7n G L and p„ G P. Likewise let be a sequence with /3„ G 
and In G Pj converging to y. Then the commutator Pnln] converges to 

[x,y]. If we can show that the former commutator lies in Fj+iPj+i then we 
are done since is normal in G so that Fj+iP,+i is a subgroup. Now the 
claim will follow from the following group theoretical lemma. □ 

Lemma 3.12 Let G be a group and let A,B be subgroups. Suppose that B 
is normal. Let Gj for j = 0,1,2,... denote the lower central series. Let 
Aj = An Gj and Bj = B n Gj. Then 

[AB,A^Bj\ C 

Proof Let a G A, 6 G -B, a G Aj,[3 G Bj. For two group elements x, |/ we 
write for y~^xy. Then 

[a6, a/5] = abaPb~^a~^P^^a^^ 
= aab'^/3b-^a-^f3~^a-^ 



a, a\ 



The two commutators are in Aj+i resp. -Bj+i alright so we only have to care 
about the rest. 

In the same fashion as before we can move the first /?-term past the two 
&-terms producing commutators in -Bj+i- It remains to consider 
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The commutator a~^hah^^ = [a~^,b] lies in C Cj^i. But on the 

other hand, since B is normal, this commutator lies in B, therefore in -Bj+i- 
The latter also is normal, hence the first factor lies in -Bj+i. The second 
commutator a/5a~^/3~^ = [a, f3] lies in [A, Bj] and likewise turns out to lie in 
Bj+i. □ 



4 The dynamical Lefschetz trace formula for 
algebraic Anosov diffeomorphisms 

Consider an algebraic Anosov diffeomorphism of a nilmanifold X = T \ G 
induced by an automorphism / of G. According to corollary |3.9| the reduced 
leafwise cohomology H^u{X) is finite dimensional. Thus the trace of the 
induced automorphism /* on this cohomology is well defined. 

Theorem 4.1 The following formula holds: 

5^(-irTr(r|^^„(X))= ^.|det(l-T,./|T:)r^ 

Here 

= sgn det(l-T,/|T;) . 

Proof Set p = g" and T = J-"". Then we have: 

J^i-lYMf* \ HUX)) = 5^(-irTr(r |i7^(p,M)) 
p p 

= 5^(-l)^Tr(AP/,|A^p) 

p 

= det(l-/*|p) 

(16) = det(l-/,|s)det(l-/,|g/p)-i. 

Now by a theorem of Nomizu it is known that 

HP{q,R) = HP{X,R) 
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for all p. Therefore by the same calculation as before: 

det(l-M0) = 5^(-irTr(r|/7^(0,M)) 

p 

= 5^(-irTr(r|i7^'(X,M)). 

p 

As / has only non-degenerate fixed points (see below), the ordinary Lefschetz 
trace formula now implies: 



(17) 



det(l - /. I s) = Yl - I ^-^) • 



f{x) = x 



For a fixed point x = F^f of / we have T = rgf{g) ^. The commutative 
diagram: 



X 



X 



X 



X 



R 



f(a)- 



induces commutative diagrams where e = F G X: 
T^X T^X g 



;i8) T./ 



T-f 



and 



nx 



TeX 



Txf 



P • 



Hence: 



^ ' ^ I w - ;i |det(l-/, |0/p)| " ^ 



/{a;) = a; 



and (|T^ asserts that: 

det(l - /. I 0) = sgn det(l - /. | g) H Fix/ . 
Together with equation (|16D this gives the formula if we remark that T^X/T^T 
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5 Dynamical Lefschetz trace formulas for gen- 
eralized algebraic Anosov maps 



In this section we will prove dynamical Lefschetz trace formulas for general- 
ized algebraic Anosov maps with respect to homogenous invariant foliations. 

Thus let / be a generalized algebraic Anosov map for the pair {G, T) as in 
the introduction and let p C g be an /,,-invariant sub Lie algebra of q. To 
p corresponds an /-invariant closed subgroup P of G. Denote by JF the 
corresponding foliation on X = F \ G. 

By assumption / maps leaves of JF into leaves of JF and hence it induces an 
endomorphism /* of H^[X). Under the isomorphism 

(19) H^jriX) = HP{p,G°^{X)) 

the operation /* corresponds to the map on Lie algebra cohomology induced 
by on p and by pullback /* on C°°(X). 

We want to define the trace of /* on the generally infinite dimensional coho- 
mology (plQl). For this we use representation theory. 

Let (p, V) be any representation of G. The pullback representation along an 
endomorphism f : G —>■ G is defined by 

r{p,v) = {pof,v). 

It is unitary if (p, V) is. For tt G G we get an equivalence class f*7c of unitary 
representations. If / : G — * G is an isomorphism, then /*7r is irreducible. In 
this case we get a bijection f* : G ^ G. 

The following lemma will be crucial 

Lemma 5.1 Let f be an endomorphism of a simply connected nilpotent Lie 
group G such that /* does not have 1 as an eigenvalue. For vr G G assume 
that there is a non-zero G -intertwining map T from f*{V^) to . Then tt 
is the trivial representation. 

Proof Let us write {p,V) for (p^,'\/f°). Let Cj be the descending central 
series. The integer A; > such that Gk+i = 1 but G^ 7^ 1 is called the length 
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of the central series. Since / is an endomorphism it preserves all the Cj and 
in particular the subgroup C^. This connected subgroup hes in the center of 
G. Since vr was supposed to be irreducible there is hence a character 

such that 

pig) = Xig) idy for all g e Ck . 
Since / preserves Ck it follows that 

(P ° /) (g) = (X ° /) (g) idy for all g E Ck ■ 
The intertwining relation p{g)T = T{pof)[g) for all g E G now implies that 
x{g)T={xof){g)T forall^^GG. 

Since T was supposed to be non-trivial this gives the relation x = ^o/ on 
Cfc. For the derivative x* '■ = LieC^ — > C it follows that 

X* = X*°f* on Cfc . 

By assumption does not have eigenvalue 1 on g and all the less on c^. 
Hence the transposed map does not have eigenvalue 1 on c^. It follows 
that X* = 0, so X is trivial. 

If A; = then = G is abelian and it follows that p = x = 1- Thus vr is 
trivial. 

If > then p factors over G/Gk which is nilpotent of length k — 1. The 
map / induces an endomorphism of this smaller group whose derivative does 
not have the eigenvalue 1. Also, T remains an intertwining map for G/Gk- 
Thus we can iterate the argument down to the abelian case A; = to reach 
the same conclusion. □ 



Recall the decomposition (y). According to lemma |2.1| the algebraic direct 
sum 



TV 

is a dense subspace of H^{X, C). 
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Definition— Theorem 5.2 Let f be a generalized algebraic Anosov map for 

(G, r). Then we have the following assertions: 

1) If f : G ^ G is surjective, hence an automorphism, then the map f* on 
IIjr{X,C) permutes the closed subspaces i7(7r)°°); 

f*{HP{p,H{7rr)) = HP{p,H{f*7rr) . 

In particular f* leaves TC invariant. The condition f*7r — tt for the sub- 
space II^{p, H{Tr)°°) to be left invariant is equivalent to tt being the trivial 
representation. Hence it is reasonable to define the trace of f* as follows: 

Tr{f*\H^{X)) := Tr(/* | H) 

:= Tr(r|^^(p,i/(7rn) 

/*7r=7r 

= Tr{f*\HP{p,R)). 

If II^{X) is finite dimensional then our definition agrees with the usual trace 
of f* as an endomorphism of a finite dimensional space. 

2) In general, if f is only an endomorphism of G, then unless tt is the trivial 
representation, 

f*(H{7r)°°) is orthogonal to H(7r)°° 

in the Hilbert space L^(X). Here 

f* : C^{X) — > C°°(X) 

is the pullback map induced by f : X ^ X . It follows that for non-trivial n 
we have: 

f*HP{p,H{nr) c'^H'^{p,H{nT) , 

where denotes the closure of the direct sum in the topology of H^{X) . 
Again this suggests to define the trace of f* by the formula 

Tr(r|i/^(X)) :=Tr(r|i7P(p,M)). 

// Hj^{X) is finite dimensional this agrees with the usual trace. 



19 



To motivate the definition of tlie trace in O furtlier, assume tliat tlie colio- 
mology spaces SP{p, i7(7r)°°) are finite dimensional and clioose bases in all 
of them. The union of these bases defines a basis of Ti. In the situation 1) 
above writing /* : ?i — > 7i as a matrix in terms of these bases, we see that 
the sum of the diagonal entries is given by: 

J2 TT{r\H%p,H{7cr)) . 

f*n=n 

The more general case 2) can be similarly motivated. 

Remark In example |5.5| below we will see that H^{X) can be infinite di- 
mensional even if J-' has dense leaves. 

Proof of |5.2| 1) Since / : X — > X is a finite covering map we have 

= • 

Note here that is again G-invariant of volume one. Thus / induces an 
isometric embedding 

/* : L^X) — > L\X) . 

In general this map is not G-equivariant. However because of the relation 

(20) Rgof* = f*o Rf^g) on (X) for all ^ e G , 

it maps irreducible subrepresentations of L'^{X) into irreducible subrepresen- 
tations. Here isomorphic subrepresentations get mapped to isomorphic ones. 
Using (|20| ) it follows that for every isotypical component -f^(vr) we have 

rHi-n) = Hirn) . 

Equivalent ly: 

/*(i7(7r)~) =i7(/*7r)~ 

under the puUback map /* : G°°(X) ^ G°°(X). Thus the map /* = (/*, /*) 
on 

iy^(X) = ^^(p,G°°(X)) 



20 



permutes the subspaces H^ip, H{tt)°°) as stated. That /*7r = vr is equivalent 
to vr being the trivial representation is a special case of lemma 

2) For endomorphisms / : G ^ G we do not have an induced map on L'^{X) 
unless / is surjective. However the pullback map on C°°{X) is all that is 
needed. Assume that /*(i7(7r)°^) is not orthogonal to H{'k)°° . Then 

p.(r(i7(7rn)^o, 

where P-,^ denotes the orthogonal projection from L^iX) onto H^n). Since 
Ptt niaps smooth vectors to smooth vectors, it follows that 

is a non-zero G- inter twining map. So the map 

S = Sof* -.M^® 1C = H{7t)'^ — > H{tt)°° = M^® 
is non-zero and satisfies 

p{g)~S = Spifig)) 

where p = is the representation on V^. Note here that: 

Rgof* = f*o Rf^g^ on (X) for all G . 

Since G acts trivially on the finite dimensional space M^, it follows that there 
is a non-zero G-intertwining map T from to itself satisfying 

p{g)T = Tp{f{g)) for all g e G . 

In other words, T is a non-zero G-intertwining map from f*{V^) to V^. 
Applying lemma p.l\ it follows that vr is the trivial representation. The re- 
maining assertions are clear. □ 

Theorem 5.3 For a generalized algebraic Anosov map f for (G, F) and a 
homogenous foliation T corresponding to p as above, we have 

dim 

^(-l)^Tr(r|^^(X))= ^ e,\det{l -TJ\T,X/T,J^)\-' 

where 

Sx = sgn det(l - T^f \ T^T) . 
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Proof The proof of theorem O apphes verbally in this more general situa- 
tion. □ 

Formula (|1]) of the introduction can now be extended. Let / be a generalized 
algebraic Anosov diffeomorphism. Denote by Qx the generalized A-eigenspace 
of on Qc and set 

0c = 0A , 0c = 0A , 0c = 0A • 

|A|>1 |A|<1 |A|=1 

Because of the relation 

these are sub Lie algebras of gc- For 0* = 0c '^^ have q^. = g* C. 
Hence there is an /^.-invariant decomposition 

= g" © 0" © 0^ 

Here 0'^ and 0" can also be characterized as the set of v E Q such that f^{v) 
tends to zero as ^ +oo resp. — oo. 

The subalgebras 0**,0",0'^ of are called the stable resp. unstable resp. 
neutral algebra of /. Let JF* be the foliation corresponding to 0*. Then we 
have 

TX = TJ^" © TJ^' © TT^ 

and hence 

T,X/T,^" = T^T' © T,^*^ . 

Using ([T8|) it follows that for a fixed point x of / on X we have 

T,J^^ © r,J^" = T^' , T,J^" = 

where T* = T*^ fl T^X and T^^ resp. T^q is the sum of the eigenspaces of 
Txf on TxX © C corresponding to eigenvalues A of absolute value |A| < 1 
resp. |A| > 1. 
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Corollary 5.4 For every generalized algebraic Anosov dijfeomorphism f of 
{G, r) the following formula holds: 



where 



£, = sgn det(l-7;/|7;") . 



The following example shows that even if the unstable foliation of a general- 
ized algebraic Anosov diffeomorphism has dense leaves, its reduced leafwise 
cohomology can be infinite dimensional. 

Example 5.5 Let G be the 3-dimensional simply connected real Heisenberg 
group and let F be its standard lattice. We can think of G as the subgroup 
of matrices 



Then F consists of those matrices [n, m, k] with integer entries. The quotient 
X = F \ G is compact. We will identify the Lie algebra 5 of G with the Lie 
algebra of matrices 




(0 a c 
0b 




in M3(M) . 



One checks that f : G ^ G defined by 



f[x, y, z] = [x + y, 2x + y,-z+{x + yf - -y{y - 1)] 



is a Lie group automorphism with inverse: 



/ y, z]^[y- X, 2x-y,x'^ - z 



^{2x - y){2x - y - 1)] . 
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The formulas show that /(F) C F and / ^(F) C F hence /(F) = F. The 
differential : g — g of / is given by 

/4|a,6,c|] = [\a + b,2a + b,^-c\] . 

Its eigenvalues are 1 + V2, 1 - V2, -1. Hence / is a generalized algebraic 
Anosov diffeomorphism. An eigenvector for A = 1 + \/2 is given by 

v = [\l,V2,0\]. 

Hence = M ■ f . As 1, \/2 are Q-linear independent the one-dimensional 
foliation JF" has dense leaves |[AGH]| Ch. IV and according to |PS|| Proposition 
1.2 the reduced cohomology H}ru{X) is infinite dimensional. Incidentally 
because of the relation 

det(l - I g) = -4 
the automorphism / of X has 4 different fixed points. 
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